32 research outputs found
Decentralized Differentially Private Without-Replacement Stochastic Gradient Descent
While machine learning has achieved remarkable results in a wide variety of
domains, the training of models often requires large datasets that may need to
be collected from different individuals. As sensitive information may be
contained in the individual's dataset, sharing training data may lead to severe
privacy concerns. Therefore, there is a compelling need to develop
privacy-aware machine learning methods, for which one effective approach is to
leverage the generic framework of differential privacy. Considering that
stochastic gradient descent (SGD) is one of the mostly adopted methods for
large-scale machine learning problems, two decentralized differentially private
SGD algorithms are proposed in this work. Particularly, we focus on SGD without
replacement due to its favorable structure for practical implementation. In
addition, both privacy and convergence analysis are provided for the proposed
algorithms. Finally, extensive experiments are performed to verify the
theoretical results and demonstrate the effectiveness of the proposed
algorithms
Resource Constrained Vehicular Edge Federated Learning with Highly Mobile Connected Vehicles
This paper proposes a vehicular edge federated learning (VEFL) solution,
where an edge server leverages highly mobile connected vehicles' (CVs') onboard
central processing units (CPUs) and local datasets to train a global model.
Convergence analysis reveals that the VEFL training loss depends on the
successful receptions of the CVs' trained models over the intermittent
vehicle-to-infrastructure (V2I) wireless links. Owing to high mobility, in the
full device participation case (FDPC), the edge server aggregates client model
parameters based on a weighted combination according to the CVs' dataset sizes
and sojourn periods, while it selects a subset of CVs in the partial device
participation case (PDPC). We then devise joint VEFL and radio access
technology (RAT) parameters optimization problems under delay, energy and cost
constraints to maximize the probability of successful reception of the locally
trained models. Considering that the optimization problem is NP-hard, we
decompose it into a VEFL parameter optimization sub-problem, given the
estimated worst-case sojourn period, delay and energy expense, and an online
RAT parameter optimization sub-problem. Finally, extensive simulations are
conducted to validate the effectiveness of the proposed solutions with a
practical 5G new radio (5G-NR) RAT under a realistic microscopic mobility
model
Distributed Learning over Networks with Graph-Attention-Based Personalization
In conventional distributed learning over a network, multiple agents
collaboratively build a common machine learning model. However, due to the
underlying non-i.i.d. data distribution among agents, the unified learning
model becomes inefficient for each agent to process its locally accessible
data. To address this problem, we propose a graph-attention-based personalized
training algorithm (GATTA) for distributed deep learning. The GATTA enables
each agent to train its local personalized model while exploiting its
correlation with neighboring nodes and utilizing their useful information for
aggregation. In particular, the personalized model in each agent is composed of
a global part and a node-specific part. By treating each agent as one node in a
graph and the node-specific parameters as its features, the benefits of the
graph attention mechanism can be inherited. Namely, instead of aggregation
based on averaging, it learns the specific weights for different neighboring
nodes without requiring prior knowledge about the graph structure or the
neighboring nodes' data distribution. Furthermore, relying on the
weight-learning procedure, we develop a communication-efficient GATTA by
skipping the transmission of information with small aggregation weights.
Additionally, we theoretically analyze the convergence properties of GATTA for
non-convex loss functions. Numerical results validate the excellent
performances of the proposed algorithms in terms of convergence and
communication cost.Comment: Accepted for publication in IEEE TSP; with supplementary details for
the derivation
Mobile MIMO Channel Prediction with ODE-RNN: a Physics-Inspired Adaptive Approach
Obtaining accurate channel state information (CSI) is crucial and challenging
for multiple-input multiple-output (MIMO) wireless communication systems.
Conventional channel estimation method cannot guarantee the accuracy of mobile
CSI while requires high signaling overhead. Through exploring the intrinsic
correlation among a set of historical CSI instances randomly obtained in a
certain communication environment, channel prediction can significantly
increase CSI accuracy and save signaling overhead. In this paper, we propose a
novel channel prediction method based on ordinary differential equation
(ODE)-recurrent neural network (RNN) for accurate and flexible mobile MIMO
channel prediction. Differing from existing works using sequential network
structures for exploring the numerical correlation between observed data, our
proposed method tries to represent the implicit physics process of path
responses changing by specially designed continuous learning network with ODE
structure. Due to the targeted design of learning network, our proposed method
fits the mathematics feature of CSI data better and enjoy higher network
interpretability. Experimental results show that the proposed learning approach
outperforms existing methods, especially for long time interval of the CSI
sequence and large channel measurement error.Comment: 7 pages, conferenc
Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees
Federated learning (FL) has emerged as a prominent distributed learning
paradigm. FL entails some pressing needs for developing novel parameter
estimation approaches with theoretical guarantees of convergence, which are
also communication efficient, differentially private and Byzantine resilient in
the heterogeneous data distribution settings. Quantization-based SGD solvers
have been widely adopted in FL and the recently proposed SIGNSGD with majority
vote shows a promising direction. However, no existing methods enjoy all the
aforementioned properties. In this paper, we propose an intuitively-simple yet
theoretically-sound method based on SIGNSGD to bridge the gap. We present
Stochastic-Sign SGD which utilizes novel stochastic-sign based gradient
compressors enabling the aforementioned properties in a unified framework. We
also present an error-feedback variant of the proposed Stochastic-Sign SGD
which further improves the learning performance in FL. We test the proposed
method with extensive experiments using deep neural networks on the MNIST
dataset and the CIFAR-10 dataset. The experimental results corroborate the
effectiveness of the proposed method
Breaking the Communication-Privacy-Accuracy Tradeoff with -Differential Privacy
We consider a federated data analytics problem in which a server coordinates
the collaborative data analysis of multiple users with privacy concerns and
limited communication capability. The commonly adopted compression schemes
introduce information loss into local data while improving communication
efficiency, and it remains an open problem whether such discrete-valued
mechanisms provide any privacy protection. In this paper, we study the local
differential privacy guarantees of discrete-valued mechanisms with finite
output space through the lens of -differential privacy (DP). More
specifically, we advance the existing literature by deriving tight -DP
guarantees for a variety of discrete-valued mechanisms, including the binomial
noise and the binomial mechanisms that are proposed for privacy preservation,
and the sign-based methods that are proposed for data compression, in
closed-form expressions. We further investigate the amplification in privacy by
sparsification and propose a ternary stochastic compressor. By leveraging
compression for privacy amplification, we improve the existing methods by
removing the dependency of accuracy (in terms of mean square error) on
communication cost in the popular use case of distributed mean estimation,
therefore breaking the three-way tradeoff between privacy, communication, and
accuracy. Finally, we discuss the Byzantine resilience of the proposed
mechanism and its application in federated learning
TernaryVote: Differentially Private, Communication Efficient, and Byzantine Resilient Distributed Optimization on Heterogeneous Data
Distributed training of deep neural networks faces three critical challenges:
privacy preservation, communication efficiency, and robustness to fault and
adversarial behaviors. Although significant research efforts have been devoted
to addressing these challenges independently, their synthesis remains less
explored. In this paper, we propose TernaryVote, which combines a ternary
compressor and the majority vote mechanism to realize differential privacy,
gradient compression, and Byzantine resilience simultaneously. We theoretically
quantify the privacy guarantee through the lens of the emerging f-differential
privacy (DP) and the Byzantine resilience of the proposed algorithm.
Particularly, in terms of privacy guarantees, compared to the existing
sign-based approach StoSign, the proposed method improves the dimension
dependence on the gradient size and enjoys privacy amplification by mini-batch
sampling while ensuring a comparable convergence rate. We also prove that
TernaryVote is robust when less than 50% of workers are blind attackers, which
matches that of SIGNSGD with majority vote. Extensive experimental results
validate the effectiveness of the proposed algorithm